Few-shot image generation via self-adaptation

ABSTRACT

One example method involves operations for receiving a request to transform an input image into a target image. Operations further include providing the input image to a machine learning model trained to adapt images. Training the machine learning model includes accessing training data having a source domain of images and a target domain of images with a target style. Training further includes using a pre-trained generative model to generate an adapted source domain of adapted images having the target style. The adapted source domain is generated by determining a rate of change for parameters of the target style, generating weighted parameters by applying a weight to each of the parameters based on their respective rate of change, and applying the weighted parameters to the source domain. Additionally, operations include using the machine learning model to generate the target image by modifying parameters of the input image using the target style.

TECHNICAL FIELD

This disclosure generally relates to image enhancement and, morespecifically, to automatically enhancing images using an adaptive model.The adaptive model transforms image content into an adaptedrepresentation of the image content, having a target style, usingfew-shot image generation.

BACKGROUND

Image processing systems are used for providing various types ofenhanced, altered, or modified images by end users who interact withimage content. Image processing systems use a number of image capturedevices, databases, or other repositories to provide image content. Forinstance, end users use cameras in mobile devices to capture images ofthemselves (e.g., selfies) or their surrounding environment. Whileconsumer devices have made capturing images easier, many end users seekto enhance their image content.

SUMMARY

Certain aspects involve methods, systems and non-transitorycomputer-readable mediums having instructions stored thereon forgenerating adapted representations of images using a machine learningmodel. In an illustrative example, an image processing system generatesadapted representations of images that more closely match a target styleof a small domain of images using few-shot image generation. Morespecifically, the image processing system receives a request totransform an input image into a target image. The image processingsystem provides the input image to a machine learning model that istrained to adapt images. Further, training the machine learning modelincludes accessing training data having a source domain that includessource images and a target domain that includes a limited number ofartistic images. The target domain includes a target style. Training themachine learning model also involves using a pre-trained generativemodel to generate an adapted source domain of adapted images thatincludes the target style. The adapted source domain is generated bydetermining a rate of change for parameters associated with the targetstyle, generating weighted parameters by applying a weight to each ofthe parameters based on the rate of change, and by applying the weightedparameters to the source domain. Additionally, the image processingsystem uses the machine learning model to generate the target image bymodifying parameters associated with the input image, for example, usingthe target style.

Other aspects describe training a machine learning model to match atarget style of a small domain of images using few-shot imagegeneration. For instance, one example method involves a processingdevice accessing training data, which includes a source domain havingsource images and a target domain having a limited number of artisticimages in a target style. The processing device performs a step forgenerating an adapted source domain that includes adapted images in thetarget style. In addition, the processing device outputs a machinelearning model that applies the target style to an input image using theadapted source domain.

These illustrative aspects are mentioned not to limit or define thedisclosure, but to provide examples to aid understanding thereof.Additional aspects are discussed in the Detailed Description, andfurther description is provided there.

BRIEF DESCRIPTION OF THE DRAWINGS

Features, aspects, and advantages of this disclosure are betterunderstood when the following Detailed Description is read withreference to the drawings.

FIG. 1 depicts an example of a computing environment for few-shot imagegeneration via self-adaptation, according to certain aspects of thisdisclosure.

FIG. 2 depicts an example of a process for transforming an input imageinto a target image using few-shot image generation via self-adaptation,according to certain aspects of this disclosure.

FIG. 3 depicts an example of a process for training a machine learningmodel to generate adapted images, e.g., as described in FIG. 2 , bytransforming source images using few-shot image generation viaself-adaptation, according to certain aspects of this disclosure.

FIG. 4 depicts another example of a process for training a machinelearning model to generate adapted images, e.g., as described in FIG. 2, by transforming source images using few-shot image generation viaself-adaptation, according to certain aspects of this disclosure.

FIG. 5 depicts yet another example of a process for training a machinelearning model to generate adapted images, e.g., as described in FIG. 2, by transforming source images using few-shot image generation viaself-adaptation, according to certain aspects of this disclosure.

FIG. 6 depicts examples of images that are used in the processes fortraining a machine learning model to transform source images usingfew-shot image generation via self-adaptation, e.g., as described inFIGS. 3-5 , and according to certain aspects of this disclosure.

FIG. 7 depicts other examples of images that are used in the processesfor training a machine learning model to transform source images usingfew-shot image generation via self-adaptation, e.g., as described inFIGS. 3-5 , and according to certain aspects of this disclosure.

FIG. 8 depicts other examples of images that are used in processes fortraining a machine learning model to transform source images usingfew-shot image generation via self-adaptation, e.g., as described inFIGS. 3-5 , and according to certain aspects of this disclosure.

FIG. 9 depicts examples of images that are used in processes fortraining a machine learning model to transform source images usingfew-shot image generation via self-adaptation, e.g., as described inFIGS. 3-5 , with plots of scaled results of such training, and accordingto certain aspects of this disclosure.

FIG. 10 depicts an example of plots having scaled results of training amachine learning model to transform source images using imagegeneration, as described in FIGS. 3-5 , and according to certain aspectsof this disclosure.

FIG. 11 depicts an example of images that are used in processes forusing a machine learning model trained to transform source images usingfew-shot image generation via self-adaptation, e.g., as described inFIG. 2 , and according to certain aspects of this disclosure.

FIG. 12 depicts examples of conventional techniques that involveoperations avoided by certain aspects of this disclosure.

FIG. 13 depicts an example of a computing system that perform certainoperations described herein, according to certain aspects of thisdisclosure.

DETAILED DESCRIPTION

Certain aspects involve using a machine learning model to performfew-shot image generation via self-adaptation by adapting a pre-trainedgenerative model using applied weights during supervised learning.Conventional solutions that employ few-shot machine learning aretypically limited to labelling or other classification tasks. Someconventional solutions include computer-based methods that use few-shottechniques to learn a classifier using a limited number of identifiedexamples (e.g., labels). Other computer-based methods include modelsthat seek to use few-shot techniques to produce image content.

But existing computer-based models that have sought to generate imagecontent with few-shot techniques typically produce suboptimal results.For example, some existing computer-based models limit the number ofparameters that are transferred to only include either a color and/or atexture. In addition, some existing computer-based models generatesimage content that is over-fitted to one or more of the limited numberof few-shot examples. For example, some existing computer-based modelssuffer from over-fitting by producing image content that looks extremelysimilar to or exactly like one or more training examples, rather thangenerating image content that maintains diversity from previousexamples. Preserving diversity when producing image content ensures thatgenerated image content includes a sufficient amount of adaptation toreflect an aesthetically-recognizable style, while creating imagery fromlearned variations that do not appear like a copy of a training example.

While consumer devices have made capturing images easier, many end userswould benefit from an ability to enhance their image content withoutthese drawbacks. Certain aspects described herein address these issuesby, for example, generating an adapted representation of an image usinga machine learning model. For example, certain aspects described hereingenerate adapted images that more closely match a target style of asmall domain of images using few-shot image generation. The followingnon-limiting examples are provided to introduce certain aspects of thisdisclosure.

In one example, an image processing system facilitates few-shot imagegeneration using self-adaptation by receiving an input with an inputimage (e.g., a request, user input, button press, graphical userinterface (GUI) selection, text input, speech-to-text input, gesture,etc.) to generate a target image. The image processing system providesthe input image to a self-supervised machine learning model that hasbeen trained to adapt images (e.g., real images, photographs, videoimages, realistic graphical images, etc.) into target images (e.g.,having a particular target style). The machine learning model transformsthe input image into a target image in the target style by modifyingparameters associated with the input image. As a result, the machinelearning model enhances an overall quality level, e.g., by generating atarget image that includes a desired aesthetic or imaging effect. Insome examples, the machine learning model creates virtual characters oravatars in a likeness of a user that is adapted to an aesthetic oroverall appearance that is native to or associated with a virtualenvironment.

In some examples, the machine learning model is trained using a sourcedomain, which includes source images and a target domain, which includesa limited number of artistic images. The target domain includes a targetstyle. Training the machine learning model further includes using apre-trained generative model to generate an adapted source domain thatincludes a set of adapted images. The set of adapted images include thetarget style. The adapted source domain is generated by determining arate of change for each of a set of parameters associated with thetarget style, generating a set of weighted parameters by applying aweight to each of the set of parameters based on the rate of change, andby applying the set of weighted parameters to the source domain.

In another example, a machine learning model is trained by adapting apre-trained generative model. For instance, the machine learning modeladapts a pre-trained generative model that is pre-trained using a largesource domain of source images. In this example, the pre-trainedgenerative model is pre-trained to generate facial images based on alarge quantity of source images. The pre-trained generative model usesan adversarial framework to identify shared parameters in the sourceimages, for example, by applying an adversarial loss.

In some examples, the shared parameters associated with source imagesare conditionally adapted to provide estimation parameters for inferringadapted images in a particular style. Further, the pre-trainedgenerative model includes a generative model, a discriminative model, oran adversarial model. And in some examples, the pre-trained generativemodel is a generative adversarial network (GAN).

In this example, the machine learning model adapts the pre-trainedgenerative model using few-shot generation via self-adaptation. Forinstance, the machine learning model is trained to adapt source imagesinto a target style. In this example, training the machine learningmodel includes determining the target style using few-shot generation(e.g. a limited number of example images) that are obtained from atarget domain. To do so, the machine learning model generates additionaldata (e.g., training images) by adapting the estimation parameters fromthe pre-trained generative model.

For example, the machine learning model uses the estimation parametersto determine weights assigned to the estimation parameters bydetermining an importance factor for each estimation parameter. Themachine learning model determines the importance factors by computing animportance measure using a scoring function (e.g., Fisher information).The machine learning model balances different losses against the Fisherinformation by applying a regularization loss to each of the estimationparameters. Further, the machine learning model applies an elasticweight consolidation (EWC) loss to the estimation parameters to avoidoverfitting. In some examples, training the machine learning modelincludes combining an output of the pre-trained generative model usingregularized importance factors.

Certain aspects provide improvements over existing software tools forediting imagery. For instance, the machine learning model describedherein takes advantage of the benefits that stem from using few-shotimages, while avoiding many of the pitfalls of existing solutions. Amongthe benefits of using few-shot images, there is a reduction in theoverall costs associated with obtaining a sufficient amount of trainingdata. In addition, since the machine learning model creates additionaltraining images using a pre-trained model, less computations arerequired of the machine learning model because the machine learningmodel applies self-adaptation techniques to an existing set ofparameters, e.g., obtained from the pre-trained model. By leveragingparameters from the pre-trained model, the machine learning model has anability to generate imagery that more closely reflects a desired targetstyle.

For example, images generated by the machine learning model includesboth low-level parameters (e.g., global colors and/or textures), as wellas high-level parameters that capture fine details of a specified targetstyle. Thus, the machine learning model generates imagery at a reducedcost, with greater computational efficiency, and while preserving adiverse set of aesthetic characteristics learned from the pre-trainedmodel. Moreover, the machine learning model avoids typical pitfalls ofexisting models, which frequently result in suboptimal imagery that isoften blurry, out-of-focus, suffers from mode collapse, are over-fitted,and are less aesthetically pleasing overall.

Example of a Computing Environment for Few-Shot Image Generation ViaSelf-Adaptation

Referring now to the drawings, FIG. 1 depicts an example of a computingenvironment 100 for few-shot image generation via self-adaptation,according to certain aspects of this disclosure. In the example of acomputing environment 100 depicted in FIG. 1 , various client devices102 access an image processing system 108 via a data network 104. Insome aspects, as in the example of a computing environment 100, theimage processing system 108 includes a pre-trained generative model 110and a machine learning model 112. In additional or alternative aspects,the pre-trained generative model 110 and machine learning model 112could be implemented in separate, independently operated computingsystems.

The image processing system 108 includes one or more devices thatprovide and execute one or more modules, engines, applications, etc. forproviding one or more digital experiences to the user. In some aspects,the image processing system 108 includes one or more processing devices,e.g., one or more servers, one or more platforms with correspondingapplication programming interfaces, cloud infrastructure, and the like.In addition, some engines are implemented using one or more servers, oneor more platforms with corresponding application programming interfaces,cloud infrastructure, and the like. The image processing system 108 usesthe one or more processing devices to execute suitable program code forperforming one or more functions. Examples of this program code includesoftware components depicted in FIG. 1 , such as the machine learningmodel 112, adaptation engine 114, parameter analysis engine 116, andtarget generation engine 118.

The image processing system 108 uses one or more of these engines toreceive an input that includes an input image. As described in detailwith respect to the various examples below, the image processing system108 uses a trained machine learning model 112 to transform the inputimage into a desired target style. In some examples, the machinelearning model 112 executes the target generation engine 118 totransform the input image into the target style. In some aspects, theimage processing 108 automatically generates produced image content inresponse to an input image.

In some aspects, the image processing system 108 trains the machinelearning model 112. For example the computing environment 100 depictedin FIG. 1 , shows the image processing system 108 including machinelearning model 112. The machine learning model 112 is trained using oneor more suitable deep learning techniques. Examples of suitable deeplearning techniques include techniques using a deep neural network (DNN)(e.g., a feed-forward neural network (FNN), a multilayer perceptron(MLP), a recurrent neural network (RNN), long-short term memory network(LSTM), independent RNN (IndRNN), etc.), a convolutional neural network(e.g., a region convolutional neural network (“R-CNN”), Fast R-CNN, orFaster R-CNN), a deep residual network (e.g., ResNet-101), etc.

In some examples, the machine learning model 112 includes a GAN, BigGAN,LapGAN, MineGAN, StyleGAN, or a deep convolutional generativeadversarial network (DCGAN). In one example, the machine learning model112 is a five-layer DCGAN. And in the example shown in FIG. 1 , themachine learning model 112 includes the adaptation engine 114, theparameter analysis engine 116, and the target generation engine 118.

In some aspects, the image processing system 108 retrieves a corpus oftraining data from an image database 106 (e.g., a source domain and atarget domain). In one example, the image processing system 108 trainsthe machine learning model 112 using the training data. For example, theimage processing system 108 uses a pre-trained generative model (e.g.,pre-trained generative model 110) to generate an abundant amount oftraining data in the form of a source domain of source images. Themachine learning model 112 accesses the corpus of training data, forexample, by obtaining the source domain from the pre-trained generativemodel 110. In some examples, the machine learning model 112 executes theadaptation engine 114 to adapt features from source images in the sourcedomain.

In some examples, the source domain includes a type of training data ora training dataset that is input into the machine learning model 112 totrain the machine learning model 112. In one example, the source domainincludes a corpus of source images. In some examples, the source domainprovides an abundant dataset having a large quantity of source images.Further, source domains include one or more shared characteristics,classifications, resolutions, semantic relationships, types, etc. Forinstance, in some examples, the source domain includes human faces,natural landscapes, and/or a particular resolution.

Likewise, the target domain is another type of training data or atraining dataset that is input into the machine learning model 112 totrain the machine learning model 112. In one example, the target domainincludes a corpus of target images. In some examples, the target domainprovides a limited dataset having a small quantity of target images. Thetarget domain includes a target style that has a particular set orsubset of distinctive visual features. For example, some target domainsinclude target images with a target style that includes an artisticstyle such as cubism, gothic, modern, neoclassic, pop, realism.

In other examples, the artistic style includes a type or collection ofartistic characters, such as avatars, Bitmojis™ emojis, game characters,virtual reality (VR) characters, or augmented reality (AR) characters,etc. In some examples, the artistic style includes a collection ofartistic works from a single artisan. The target domain also includesshared characteristics, classifications, resolutions, semanticrelationships, or types, etc. The target style includes one or moreparameters that are aesthetically identifiable or otherwise visuallyassociated with the few-shot images (e.g., an artistic collection).

The machine learning model 112 is trained to adapt source images into atarget style. The machine learning model 112 is trained by adapting thepre-trained generative model 110. For instance, the machine learningmodel 112 adapts the pre-trained generative model 110 that ispre-trained using a source domain of source images from image database106. The pre-trained generative model 110 is trained to generate facialimages using the source domain. In some examples, pre-trained generativemodel 110 uses a StyleGAN network architecture.

In one example, the pre-trained generative model 110 uses an adversarialframework to identify shared parameters in the source images. Forexample, the pre-trained generative model 110 identifies parameters byapplying an adversarial loss. In this example, the pre-trainedgenerative model 110 computes adversarial loss using the followingexpression.

$L_{adv} = {{\min\limits_{G}\max\limits_{D}{\varepsilon_{x\sim{P_{data}(x)}}\left\lbrack {\log{D(x)}} \right\rbrack}} + {{\varepsilon_{z}}_{\sim{P_{z}(z)}}\left\lbrack {\log\left( {1 - {D\left( {G(z)} \right)}} \right)} \right\rbrack}}$Here, L_(adv) represents the adversarial loss, G represents a generatorfor the pre-trained generative model 110, D represents a discriminatorfor the pre-trained generative model 110,

$L_{adv}^{\prime} = {{\underset{G\prime}{\min}\max\limits_{D\prime}{\varepsilon_{x \sim {P_{data}(x)}}\left\lbrack {\log{D^{\prime}(x)}} \right\rbrack}} + {\varepsilon_{z \sim {P_{z}(z)}}\left\lbrack {\log\left( {1 - {D^{\prime}\left( {G^{\prime}(z)} \right)}} \right)} \right\rbrack}}$represents a minmax value function for G and D, ε_(x˜P) _(data) _((x))represents an expected value over all real data instances (e.g.,x˜P_(data)(x)), ε_(z˜P) _(z) _((z)) represents an expected value overall random inputs to the generator G (e.g., z˜P_(z)(z)), P_(data)(x)represents a distribution of noise associated with a subset of sourceimages from the source domain, and P_(z) (z) represents a distributionof noise among a set of parameters of the source domain.

Training the machine learning model 112 includes generating additionaltraining data. Further, the machine learning model 112 determines atarget style of the target domain. For example, machine learning model112 uses the adaptation engine 114 to generate additional data (e.g.,abundant training images) by adapting the set of parameters associatedwith the source domain. And in this example, the machine learning model112 adapts the pre-trained generative model 110 using few-shotgeneration via self-adaptation.

For instance, the machine learning model 112 executes the adaptationengine 114 that also uses an adversarial framework to identify thetarget style, e.g., which includes shared parameters among the targetimages. Specifically, the adaptation engine 114 fine-tunes thepre-trained generative model 110 by applying an adversarial loss to thesource domain in a style of the target domain to generate more trainingdata. Like the pre-trained generative model 110, the adaptation engine114 computes the adversarial loss using the following expression.

$\min\limits_{G}\max\limits_{D}$But here, L′_(adv) represents the adversarial loss,

$\min\limits_{G^{\prime}}\underset{D^{\prime}}{\max}$represents a minmax value function for G′ and D′, ε_(x˜P) _(data) _((x))represents an expected value produced over all real data instances(e.g., x˜P_(data)(x)), ε_(z˜P) _(z) _((z)) represents an expected valueover all random inputs to the generator (e.g., z˜P_(z)(z)), G′represents a generator associated with an adapted generative modelproduced by the adaptation engine 114, D′ represents a discriminatorthat is associated with the adaptation engine 114, P_(data)(x)represents a distribution of noise associated with the few-show, limitednumber of target images (e.g., the target domain), and P_(z)(z)represents a distribution of noise for a set of parameters associatedwith the target domain. The adaptation engine 114 generates adaptedimages with diversity preserved from the source images, e.g., thatinclude an overall aesthetic appearance that is substantially similar tothe target style. The adaptation engine 114 provides the adapted imagesto the parameter analysis engine 116.

The machine learning model 112 executes the parameter analysis engine116, which obtains the adapted images from the adaptation engine 114.For example, the parameter analysis engine 116 uses parameters from thepre-trained generative model 110 and the adaptation engine 114 todetermine an average rate of change for weights assigned to theparameters. The parameter analysis engine 116 analyzes weights assignedto parameters associated with source domain and parameters associatedwith the target domain using data from the pre-trained generative model110 and the adaptation engine 114, respectively. In some examples, themachine learning model 112 executes the parameter analysis engine 116 intandem (e.g., substantially simultaneously) with the adaptation engine114.

For instance, the parameter analysis engine 116 determines the averagerate of change for weights assigned to the parameters using thefollowing expression.

$\bigtriangleup = {\frac{1}{N}{\sum}_{i}\frac{\left| {\theta_{G^{\prime},i} - \theta_{G,i}} \right|}{\theta_{G,i}} \times 100\%}$Here, G is obtained from pre-trained generative model 110, G′ isobtained from the adaptation engine 114, N represents a number ofparameters, θ_(G,i) represents an i-th parameter in the model G, andθ_(G′,i) represents an i-th parameter in the adapted model G′ providedby the adaptation engine 114. In some examples, the parameter analysisengine 116 computes the average rate of change for the weights for eachparameter at each convolutional layer.

The machine learning model 112 also uses the parameter analysis engine116 to determine an importance factor for each estimation parameter. Forinstance, the parameter analysis engine 116 estimates an overallimportance of parameters using learned values. The parameter analysisengine 116 determines the importance factors by computing an importancemeasure using a scoring function (e.g., by calculating Fisherinformation). In one example, the parameter analysis engine 116 computesthe Fisher information F for learned values of parameters θ_(s) usingthe following expression.

$F = {E\left\lbrack {- {\frac{\partial^{2}}{\partial\theta_{s}^{2}}{\mathcal{L}\left( {X{❘\theta_{s}}} \right)}}} \right\rbrack}$Here,

(X|θ_(s)) is a log-likelihood function that is a computationalequivalent of a binary cross-entropy loss, e.g., using an output of adiscriminator, where θ_(s) represents the learned values of weights ofparameters from the pre-trained generative model 110 G, and X representsan amount of generative training images that is based on the learnedvalues θ_(s).

The machine learning model 112 balances different losses against theFisher information by applying a regularization loss to each of theestimation parameters. For instance, the parameter analysis engine 116uses the Fisher information F calculated above to determine aregularization loss. Further, the parameter analysis engine 116 appliesthe regularization loss to penalize a weight change that occurs duringan adaptation to a target domain. To do so, the parameter analysisengine 116 uses the following expression.

$L_{adapt} = {L_{adv} + {\lambda{\sum\limits_{i}{F_{i}\left( {\theta_{i} - \theta_{S,i}} \right)}^{2}}}}$Here, L_(adapt) represents the regularization loss, which is anadaptation loss that includes the adversarial loss described above(e.g., L_(adv)), F_(i) represents a Fisher information matrix, θrepresents values for parameters of the target style, θ_(s) representsthe learned values of weights of parameters from the pre-trainedgenerative model 110 G, and λ represents a regularization weight thatbalances various losses, and the entire term λΣ_(i)F_(i)(θ_(i)−θ_(S,i))²represents an EWC loss, which is applied to parameters to avoidoverfitting.

Some aspects of the computing environment 100 include client devices102. For instance, some client devices 102 are operated by cliententities (e.g., commercial entities or content providers) that request atransformation of images using techniques discussed herein. Suchrequests are performed by sending images directly to the imageprocessing system 108. In another example, the client devices 102 areoperated by end users that desire to enhance various image content ofinterest. For instance, end users send requests for a transformation ofan image, collection of images, video, vlog, selfie, or social mediastory, etc.

Examples of a client device 102 include, but are not limited to, apersonal computer, a laptop, a tablet, a desktop, a server, a mobiledevice, a smartphone, a processing unit, any combination of thesedevices, or any other suitable device having one or more processors. Auser of a client device 102 uses various products, applications, orservices supported by the image processing system 108 via the datanetwork 104.

Each of the client devices 102 is communicatively coupled to the imageprocessing system 108 via the data network 104. Examples of the datanetwork 104 include, but are not limited to, internet, local areanetwork (“LAN”), wireless area network, wired area network, wide areanetwork, and the like.

In the example of the computing environment 100, the image processingsystem 108 depicted in FIG. 1 executes the machine learning model 112 togenerate target images. The image processing system 108 outputs thetarget images to a requesting client device 102. In one example,outputting target images includes encoding and transmitting the targetimage to the client device 102. In some examples, the image processingsystem 108 encodes the target image using any suitable image format(e.g., AVI, BGP, BMP, CGM, Exif, FLV, F4V, GIF, HDR, HEIF, HEVC, JPEG,MPEG, MP4, PNG, PPM, RIFF, SVG, TIFF, VC-1, VVC, WebP, XPS, or any othersuitable imaging format). In other examples, outputting the target imageincludes electronic storage on a memory associated with the computingenvironment 100.

Examples of Processes for Few-Shot Image Generation Via Self-Adaptation

FIG. 2 is an example of a process 200 for transforming an input imageinto a target using few-shot image generation via self-adaptation,according to certain aspects of this disclosure. One or more operationsdescribed with respect to FIG. 2 transform an input image using an imageprocessing system (e.g., image processing system 108). The imageprocessing system 108 executes a trained machine learning model (e.g.,machine learning model 112) to produce target images according certainaspects discussed herein. One or more processing devices (e.g.,computing environment 100) implement operations depicted in FIG. 2 byexecuting suitable program code (e.g., machine learning model 112). Forillustrative purposes, the process 200 is described with reference tocertain examples depicted in the figures. Other implementations,however, are possible.

At block 202, the process 200 involves receiving a request to transforman input image into a target image. Examples of received requests aredescribed in further detail below with respect to FIG. 9 . In oneexample, image processing system 108 receives a request to transform theinput image from a client device (e.g., client device 102). The requestto transform the input image is sent from any of the client devices 102described above.

In some examples, a user of the client device 102 requeststransformation of an input image that includes a desired target style.Further, the target style includes a particular artistic style. In someexamples, the input image is a real image (e.g., a photograph). In oneexample, the client device 102 sends a request to transform an inputimage that is stored locally at the client device 102 or remotely (e.g.,at an image database 106). The request for transformation of the inputimage includes generating a target image based on the input image. Insome examples, the target image includes a graphical representation ofthe input image.

For instance, the request from the client device 102 includes a capturedimage (e.g., a photograph captured by a camera integrated in the clientdevice 102). The request is entered in an application being executed onthe client device 102 (e.g., via a GUI). In some examples, theapplication allows a user to select a desired target style from a listor menu of available target styles. For example, the client device 102sends a request to transform a captured image of a self-portrait (e.g.,a selfie) into a desired target style. In this example, the desiredtarget style includes an avatar, Bitmoji™ emoji, game character, VRcharacter, AR character, or another desired artistic target style.

At block 204, the process 200 involves providing the input image to thetrained machine learning model 112 that is trained to adapt images. Forinstance, the image processing system 108 sends the input images to themachine learning model 112. The image processing system 108 executes themachine learning model 112, which is trained to adapt images accordingto any of the techniques described herein. The machine learning model112 identifies parameters associated with the input image and targetstyle. For example, the machine learning model 112 determines parametersshared by the input image and target style.

At block 206, the process 200 involves generating, using the trainedmachine learning model 112, the target image by modifying one or moreparameters of the input image using a target style. For example, thetrained machine learning model 112 generates the target image using anyof the techniques described herein. In some examples, the target imageis a graphical representation of the image. For instance, the trainedmachine learning model 112 generates a graphical representation thatincludes one or more shared characteristics based on parametersassociated with the input image and the target style, e.g., targetparameters of a target domain.

In some examples, the graphical representation includes an avatar,Bitmoji™ emoji, game character, virtual character, AR character,landscape, still life, impression, wildlife, animal, portrait, etc. Andin some examples, the trained machine learning model 112 provides thetarget image to image processing system 108 or the client device 102.For instance, the client device 102 receives the target image inresponse to the request. Further, the client device 102 renders thetarget image on a display, for example, using the application describedabove. In other examples, the trained machine learning model 112 outputsthe target to a non-display device such as the image database 106.

FIG. 3 is an example of a process 300 for training a machine learningmodel to generate adapted images, e.g., by transforming source imagesusing few-shot image generation via self-adaptation, according tocertain aspects of this disclosure. Operations described with respect toFIG. 3 transform source images using an image processing system (e.g.,image processing system 108). The image processing system 108 executes atrained machine learning model (e.g., machine learning model 112) toproduce adapted images with a target style according certain aspectsdiscussed herein. One or more processing devices (e.g., computingenvironment 100) implement operations depicted in FIG. 3 by executingsuitable program code (e.g., machine learning model 112). Forillustrative purposes, process 300 is described with reference toexamples depicted in the figures; however, other implementations arepossible.

At block 302, the process 300 involves accessing a source domain thatincludes source images and a target domain that includes a limitednumber of artistic images with a target style. For instance, the imageprocessing system 108 accesses the source domain that includes thesource images. In some examples, the image processing system 108accesses the source domain by requesting, retrieving, or otherwiseobtaining the source domain from a remote computing device or repositorysuch as the image database 106. Further, in some examples, the imageprocessing system 108, pre-trained generative model 110, the machinelearning model 112, or a combination of these accesses the sourcedomain.

As described in greater detail below, with respect to FIGS. 6 and 9 ,the source domain includes source images that depict a large quantity oftraining images. In some examples, such abundant training data isrelated to images contained in the target domain. In one example, thesource domain is semantically related to the target domain. In anotherexample, the source domain includes real faces that are semanticallyrelated to faces of the target domain (e.g., emoji faces, cartoon faces,or caricatures, etc.). In this example, the source domain includes facesgenerated using an abundant training dataset previously provided to thepre-trained generative model 110. Similarly, the image processing system108 accesses the target domain.

In this example, the target domain includes a limited number of artisticimages that share a target style. Further, the limited number ofartistic images are few-shot images that are used to generate anabundant training dataset. In some examples, a number of few-shot imagesare limited to a predetermined threshold value. For instance, in oneexample, the number of few-shot images includes 10 or fewer images. Theimage processing system 108 provides the source domain and the targetdomain to the machine learning model 112. Examples of target domainsthat include target images having shared parameters are described ingreater detail below, e.g., with respect to FIGS. 6-8, 11, and 12 .

At block 304, the process 300 involves generating an adapted sourcedomain that includes adapted images with the target style. For instance,the machine learning model 112 is trained according to any of thetechniques described herein. Specifically, the machine learning model112 uses weights that are determined based on information associatedwith the generated source images to adapt the pre-trained generativemodel 110.

For instance, the machine learning model 112 obtains weights associatedwith parameters identified by the pre-trained generative model 110. Forexample, the weights are computed during a generation of the sourceimages, e.g., when the pre-trained generative model 110 applied anadversarial loss to the source domain. The machine learning model 112adapts the pre-trained generative model 110 by fine-tuning these weightsusing data obtained from an adversarial loss that is applied to thetarget domain.

For instance, the machine learning model 112 determines an importancefactor for each parameters. In one example, the machine learning model112 determines the importance factors by computing Fisher informationafter each convolutional layer. In some examples, the machine learningmodel 112 applies a regularization weight to balance various lossesduring training. For instance, the machine learning model 112 adds aregularization loss that penalizes weight changes during the adaptationof source images to a target style of the target domain. Further, insome examples, the machine learning model 112 applies an EWC loss to thechanges of the weights of each parameter during the adaptation of theseparameters.

At block 306, the process 300 involves outputting a trained machinelearning model (e.g., machine learning model 112) configured to generatea representation of an input image in the target style. For instance,the machine learning model 112 is outputted to and/or stored in theimage processing system 108. And in some examples, the machine learningmodel 112 is output once a convergence point is reached. For instance,the machine learning model 112 determines that an EWC loss has reached aconvergence point that is associated with each of the parameters. Insome examples, the convergence point includes a threshold change in anamount or percentage of an iterative EWC loss. In additional oralternative examples, the convergence point includes an EWC loss thatreflects an amount of saturation, e.g., indicating that the weights ofparameters are substantially unchanged over time.

FIG. 4 is an example of a process 400 for training a machine learningmodel to generate adapted images, e.g., by transforming source imagesusing few-shot image generation via self-adaptation, according tocertain aspects of this disclosure. One or more operations describedwith respect to FIG. 4 transform source images using an image processingsystem (e.g., image processing system 108). Image processing system 108executes a trained machine learning model (e.g., machine learning model112) to generate adapted images with a target style according to certainaspects discussed herein. One or more processing devices (e.g.,computing environment 100) implement operations depicted in FIG. 4 byexecuting suitable program code (e.g., machine learning model 112). Forillustrative purposes, the process 400 is described with reference tocertain examples depicted in the figures. Other implementations,however, are possible.

At block 402, the process 400 involves accessing a source domain thatincludes source images and a target domain that includes a limitednumber of artistic images with a target style. The image processingsystem 108 accesses the source domain having the source images, forexample, by requesting, retrieving, or otherwise obtaining the sourcedomain from a remote computing device or repository such as the imagedatabase 106. In some examples, the image processing system 108,pre-trained generative model 110, the machine learning model 112, or acombination of these accesses the source domain. The image processingsystem 108 accesses the source domain according to any of the techniquesdescribed herein.

At block 404, the process 400 involves determining a rate of change foreach of a set of parameters associated with the target style. Forinstance, the machine learning model 112 computes Fisher information foreach of the set of parameters during an adaptation to the target style.In some examples, the rate of change for each of the weights iscalculated for each convolutional layer. Further, the rate of change forthese parameters is calculated according to any of the techniquesdescribed herein.

At block 406, the process 400 involves generating a set of weightedparameters by applying a weight to each of the set of parameters basedon the rate of change. For instance, the machine learning model 112 usesthe Fisher information obtained at block 404 to generate a set ofweighted parameters to be applied for each of the set of parametersduring an adaptation to the target style. The machine learning model 112does so by using the Fisher information as an importance measure orfactor. For example, the Fisher information serves as a quantifiableimportance factor that is selectively applied to the set of parametersas a regularization weight during an adaption to the target style. Insome examples, such an importance factor is applied to each of theweighted parameters for each convolutional layer. The importance factoris used to generate or regenerate weighted parameters using any of thetechniques described herein.

At block 408, the process 400 involves applying the set of weightedparameters to the source domain. For example, the machine learning model112 balances different losses against the Fisher information obtainedfrom block 406 by applying a regularization loss to each of the weightedparameters. For instance, the machine learning model 112 uses the Fisherinformation to determine the regularization loss.

The machine learning model 112 determines and applies the regularizationloss to each of the weighted parameters. The regularization loss is usedto penalize the rate of change determined at block 404, which occursduring an adaptation to the target style. In doing so, the machinelearning model 112 preserves valuable diversity obtained from the sourcedomain. In some examples, the machine learning model 112 applies the setof weighted parameters to the source domain according to any of thetechniques described herein.

At block 410, the process 400 involves generating an adapted sourcedomain that includes adapted images with the target style using theapplied set of weighted parameters. For instance, the machine learningmodel 112 uses the applied weights from block 408 to adapt source imagesof the source domain, transforming the source images into adaptedimages. In some examples, the machine learning model 112 generates theadapted source domain using any of the techniques described herein.

At block 412, the process 400 involves outputting a trained machinelearning model (e.g., machine learning model 112) configured to generatea representation of an input image in the target style. For instance,the machine learning model 112 is outputted to and/or stored in theimage processing system 108. And in some examples, the machine learningmodel 112 is output once a convergence point is reached. For instance,the machine learning model 112 determines that an EWC loss has reached aconvergence point that is associated with each of the parameters. Insome examples, the convergence point includes a threshold change in anamount or percentage of an iterative EWC loss. In additional oralternative examples, the convergence point includes an EWC loss thatreflects an amount of saturation, e.g., indicating that the weights ofparameters are substantially unchanged over time.

FIG. 5 is another example of a process 500 for training a machinelearning model to generate adapted images, e.g., by transforming sourceimages using few-shot image generation via self-adaptation, according tocertain aspects of this disclosure. Operations described with respect toFIG. 5 transform source images using an image processing system (e.g.,image processing system 108). The image processing system 108 executes atrained machine learning model (e.g., machine learning model 112) toproduce adapted images with a target style according certain aspectsdiscussed herein. One or more processing devices (e.g., computingenvironment 100) implement operations depicted in FIG. 5 by executingsuitable program code (e.g., machine learning model 112). Forillustrative purposes, the process 500 is described with reference tocertain examples depicted in the figures. Other implementations,however, are possible.

At block 502, the process 500 involves accessing a source domain thatincludes source images and a target domain that includes a limitednumber of artistic images with a target style. The image processingsystem 108 accesses the source domain having the source images, forexample, by requesting, retrieving, or otherwise obtaining the sourcedomain from a remote computing device or repository such as the imagedatabase 106. In some examples, the image processing system 108,pre-trained generative model 110, the machine learning model 112, or acombination of these accesses the source domain. The image processingsystem 108 accesses the source domain according to any of the techniquesdescribed herein.

At block 504, the process 500 involves convolving, iteratively, a set oflayers associated with the source images using a pre-trained generativemodel and an adapted generative model. The machine learning model 112gradually adapts features and/or parameters associated with the sourceimages over time, e.g., using a set of convolutional layers. Forexample, machine learning model 112 convolves each layer by applying oneor more weighted parameters to the source domain provided by thepre-trained generative model. The machine learning model 112 uses theiterative convolutional adaptation to fine-tune weighted parametersusing a distribution of noise variables associated with the targetdomain.

At block 506, the process 500 involves determining a rate of change foreach of a set of parameters associated with the target style. Forinstance, the machine learning model 112 computes Fisher information foreach of the set of parameters during an adaptation to the target style.In some examples, the rate of change for each of the weights iscalculated for each convolutional layer. Further, the rate of change foreach of the set of parameters associated with the target style iscalculated according to any of the techniques described herein.

At block 508, the process 500 involves computing an average rate ofchange for a set of weighted parameters using applied weights associatedwith each of the set of parameters. For example, the machine learningmodel 112 determines the average rate of change of the applied weightsfor each of the iterative convolutional layers. In some examples, one ormore parameters are selectively omitted. For instance, in some examples,a bias, a normalization parameter, or another selected parameter isomitted from a computation of the average rate of change. In someexamples, the average rate of change for the set of weighted parametersis calculated using any of the techniques described herein.

At block 510, the process 500 involves regularizing the set of weightedparameters to create an importance factor associated with each of theset of weighted parameters. For instance, the machine learning model 112uses the average rate of change for the weighted parameters from block508 to generate a set of weighted parameters to be applied for each ofthe set of parameters during an adaptation to the target style. Themachine learning model 112 does so by using the Fisher information as animportance measure or factor. For example, the Fisher information servesas a quantifiable importance factor that is selectively applied to theset of parameters as a regularization weight during an adaption to thetarget style. In some examples, the importance factor is applied to eachweighted parameter for each convolutional layer. Further, the importancefactor is used to regularize the set of weighted parameters using any ofthe techniques described herein.

At block 512, the process 500 involves generating an adapted sourcedomain that includes adapted images with the target style using theregularized set of weighted parameters. For instance, the machinelearning model 112 uses the regularized importance factors associatedwith the weighted parameters from block 510 to adapt source images intoan adapted source domain of adapted images. In some examples, themachine learning model 112 generates the adapted source domain using anyof the techniques described herein.

At block 514, the process 500 involves outputting a trained machinelearning model (e.g., machine learning model 112) configured to generatea representation of an input image in the target style. For instance,the machine learning model 112 is outputted to and/or stored in theimage processing system 108. And in some examples, the machine learningmodel 112 is output once a convergence point is reached. For instance,the machine learning model 112 determines that an EWC loss has reached aconvergence point that is associated with each of the parameters. Insome examples, the convergence point includes a threshold change in anamount or percentage of an iterative EWC loss. In additional oralternative examples, the convergence point is an EWC loss that reflectsan amount of saturation, e.g., indicating that the weights of parametersare substantially unchanged over time.

Examples of Training Imagery Used for Training Model to Transform SourceImages Using Few-Shot Image Generation

The following example is provided to illustrate a potential applicationof the operations described above. FIG. 6 depicts examples 600 of imagesthat are used in the processes for training a machine learning model totransform source images using few-shot image generation viaself-adaptation, according to certain aspects of this disclosure. Inparticular, FIG. 6 depicts simplified examples 600 of images that areused in the process for training a machine learning model fortransforming a large source domain into a target style of a targetdomain (e.g., target domain 602) using few-shot image generation viaself-adaptation. Operations described with respect to FIG. 6 transformsource images using an image processing system (e.g., image processingsystem 108). The image processing system 108 trains a machine learningmodel (e.g., machine learning model 112) to generate adapted images(e.g., adapted images 604) with a target style according to certainaspects discussed herein. One or more processing devices (e.g.,computing environment 100) implement operations depicted in FIG. 6 byexecuting suitable program code. For illustrative purposes, the example600 is described with reference to certain examples depicted in thefigures. Other implementations, however, are possible.

In this example, the source domain includes a subset of source images606 that depict illustrative examples provided by a pre-trainedgenerative model (e.g., via generator G 608 for a pre-trained generativemodel) to the machine learning model 112. Although generator G 608 isdepicted as having five convolutional layers, it should be appreciatedthat generator G 608 includes any number of or type of convolutionallayers. In some examples, the generator G 608 for a pre-trainedgenerative model includes all of the capabilities described above, e.g.,with respect to the pre-trained generative model 110 of FIG. 1 .

The machine learning model 112 uses the source images 606 to generate aset of adapted images 604. For instance, the machine learning model 112generates the set of adapted images 604 in a target style associatedwith the target domain 602. And in this example, the machine learningmodel 112 uses an adapted generator G′ 610 for an adapted generativemodel to generate the set of adapted images 604 in the target style.While the adapted generator G′ 610 is depicted having five convolutionallayers, it should be appreciated that the adapted generator G′ 610 alsoincludes any suitable number of or type of convolutional layers. In someexamples, the adapted generator G′ 610 for the adapted generative modelincludes all of the capabilities described above, e.g., with respect tothe adaptation engine 114 of FIG. 1 .

FIG. 7 depicts other examples 700 of images that are used in theprocesses for training a machine learning model to transform sourceimages using few-shot image generation via self-adaptation, according tocertain aspects of this disclosure. In particular, FIG. 7 depicts asimplified example 700 of a diagram of training a machine learning modelfor transforming a source domain into a target style of a target domainusing few-shot image generation via self-adaptation. One or moreoperations described with respect to FIG. 7 transform source imagesusing an image processing system (e.g., image processing system 108).The image processing system 108 trains a machine learning model (e.g.,machine learning model 112) to generate adapted images to provideadditional training images (e.g., generated training images 704) using atarget style (e.g., of few-shot image 702) according to certain aspectsdiscussed herein. One or more processing devices (e.g., computingenvironment 100) implement operations depicted in FIG. 7 by executingsuitable program code (e.g., machine learning model 112). Forillustrative purposes, the example 700 is described with reference tocertain examples depicted in the figures. Other implementations,however, are possible.

In this examples, the machine learning model 112 generates additionaltraining images (e.g., generated training images 704) using the targetstyle associated with a few-shot image 702. For instance, the machinelearning model 112 employs an adversarial framework to identifyparameters associated with the target style according to any of thetechniques described herein. Further, the machine learning model 112applies an adversarial loss to a source domain in the target style toobtain the generated training images 704. As described above, themachine learning model 112 computes the adversarial loss to generate thegenerated training images 704 while preserving diversity from the sourceimages.

In this example, the machine learning model 112 uses shared parametersbetween the source domain and the exemplary few-shot image 702 togenerate the generated training images 704 that include an aestheticappearance of the target style. For example, the generated trainingimages 704 include a substantially similar identity, e.g., havingvarious aesthetic features in common with the few-shot image 702.However, the generated training images 704 also include a diverse set ofvariations.

For instance, some of the generated training images 704 includediversity that is reflected by a youthful overall appearance, additionalaccessories (e.g., glasses, raised collars, or necklaces, etc.),alterations of skin tones, differences in hairstyles (e.g., parts,bangs, gray or streaks, etc.), facial features (e.g., dimples, toothysmiles, or an eyebrow thickness, etc.), facial expressions, poses, orother variations in appearance, etc. While the generated training images704 are depicted in FIG. 7 with respect to facial images, it should beappreciated that additional training images are generated for anysuitable type of images. For instance, some generated training images704 include landscapes, still life, impressions, wildlife, animals, orportraits, etc.

FIG. 8 depicts other examples 800 of images that are used in theprocesses for training a machine learning model to transform sourceimages using few-shot image generation via self-adaptation, according tocertain aspects of this disclosure. In particular, FIG. 8 depicts anexample 800 of a diagram of training a machine learning model fortransforming a large source domain into a target style of a targetdomain (e.g., a style associated with few-shot images 802). One or moreoperations described with respect to FIG. 8 transform source imagesusing an image processing system (e.g., image processing system 108).The image processing system 108 trains a machine learning model (e.g.,machine learning model 112) to generate adapted images (e.g., adaptedimages 804) with a target style according to certain aspects discussedherein. One or more processing devices (e.g., computing environment 100)implement operations depicted in FIG. 8 by executing suitable programcode (e.g., machine learning model 112). For illustrative purposes, theexample 800 is described with reference to certain examples depicted inthe figures. Other implementations, however, are possible.

In this example, the machine learning model 112 uses shared parametersbetween a source domain and the exemplary few-shot images 802 togenerate adapted images 804. The machine learning model 112 generatesadapted images 804 that include an aesthetic appearance of a targetstyle associated with the few-shot images 802. For example, the adaptedimages 804 may include one or more target parameters of the target stylethat are associated with the few-shot images 802. The adapted images 804include a similar overall aesthetic, e.g., sharing some features withthe caricatures depicted in the few-shot images 802, while preservingdiversity from the source domain.

FIG. 9 depicts examples 900 of images that are used in processes fortraining a machine learning model to transform source images usingfew-shot image generation via self-adaptation with plots of scaledresults of such training, according to certain aspects of thisdisclosure. In particular, FIG. 9 depicts examples 900 of images thatare used in processes for training a machine learning model fortransforming a large source domain (e.g., that includes source images906) into a target style of a target domain (e.g., adapted images 904)using few-shot image generation via self-adaptation. One or moreoperations described with respect to FIG. 9 transform source imagesusing an image processing system (e.g., image processing system 108).The image processing system 108 trains a machine learning model (e.g.,machine learning model 112) to generate adapted images (e.g., adaptedimages 904) with a target style according to certain aspects discussedherein. One or more processing devices (e.g., computing environment 100)implement operations depicted in FIG. 9 by executing suitable programcode (e.g., machine learning model 112). For illustrative purposes, theexample 900 is described with reference to certain examples depicted inthe figures. Other implementations, however, are possible.

In this example, the machine learning model 112 generates adapted images904 using a target style associated with few-shot images. For instance,machine learning model 112 generates adapted images 904 from sourceimages 906 into the target style according to any of the techniquesdescribed herein. The source domain includes a subset of source images906 that depict certain examples provided by a generator G 908 for apre-trained generative model. The generator G 908 includes all of thecapabilities described above, e.g., with respect to pre-trainedgenerative model 110 and generator G 608 of FIGS. 1 and 6 ,respectively.

The machine learning model 112 generates the set of adapted images 904in a target style, for example, using an adapted generator G′ 910 for anadapted generative model. In some examples, adapted generator G′ 910includes all of the capabilities described above, e.g., with respect tothe adaptation engine 114 and adapted generator G′ 610 of FIGS. 1 and 6, respectively. Additionally, while the generators G 908 and G′ 910 aredepicted as having five convolutional layers, it should be appreciatedthat each of the generators G 908 and G′ 910 includes any suitablenumber of or type of convolutional layers.

In this example, FIG. 9 also depicts statistical information derivedfrom an analysis of the generators G 908 and G′ 910. In particular, arate of changes of weights 902 and Fisher information 912 is shown forthe generators G 908 and G′ 910 over five convolutional layers (e.g.,Conv0-Conv4), which are implemented in the form of two substantiallysimilar five-layer DCGAN networks. The rate of changes of weights 902indicates a comparative rate of changes for weights at differentconvolutional layers between the generators G 908 and G′ 910. The Fisherinformation 912 reflects an average Fisher information for weights atdifferent convolutional layers in the generator G 908.

The rate of changes of weights 902 and Fisher information 912 indicatethat the weights change the least for the adapted generator G′ 910 inthe last convolutional layer, Conv4. In some examples, latterconvolutional layers, e.g., the last convolutional layer Conv4, is usedby the adapted generator G′ 910 to synthesize low-level features sharedacross domains. Further, in some examples, low-level features includecolors, textures, edges, or contours, etc. And in some examples, suchlow-level features are shared across domains, thereby allowing theadapted generator G′ 910 to preserve important features by adaptingthese low-level features in later convolutional layers.

FIG. 10 depicts an example 1000 of plots having scaled results oftraining a machine learning model to transform source images usingfew-shot image generation via self-adaptation, according to certainaspects of this disclosure. Specifically, the example 1000 depictscertain advantages of using few-shot image generation viaself-adaptation. One or more operations described with respect to FIG.10 transform source images using an image processing system (e.g., imageprocessing system 108). The image processing system 108 trains a machinelearning model (e.g., machine learning model 112) to generate adaptedimages with a target style according certain aspects discussed herein.One or more processing devices (e.g., computing environment 100)implement operations depicted in FIG. 10 by executing suitable programcode (e.g., machine learning model 112). For illustrative purposes, theexample 1000 is described with reference to certain examples depicted inthe figures. Other implementations, however, are possible.

The example 1000 shows an overall effectiveness of EWC loss. In thisexample, a plot of EWC loss is shown without regularization 1002 and aplot of EWC loss is shown with regularization 1004. Both plots showimage generation results for an artistic target domain adapted from asource domain of real faces, and loss values in each of the two plotsare amplified by the same scalar for better illustration. Further, thesource domain included images having the same size of 256×256 pixels. Inthis example, each of the plots without regularization 1002 and withregularization 1004 was generated during target adaptation.

As described above with respect to FIG. 1 , regularization loss iscalculated, e.g., by the parameter analysis engine 116 using thefollowing expression.

$L_{adapt} = {L_{adv} + {\lambda{\sum\limits_{i}{F_{i}\left( {\theta_{i} - \theta_{S,i}} \right)}^{2}}}}$Here, the regularization loss (e.g., the adaptation loss L_(adapt)) iscalculated by adding the adversarial loss described above (e.g.,L_(adv)), F_(i) represents a Fisher information matrix, θ represents thevalues for parameters associated with the target style, θ_(s) representsthe learned values of weights of parameters, and the EWC loss isrepresented by the second term λΣ_(i)F_(i)(θ_(i)−θ_(S,i))². But the plotwithout regularization 1002 was generated by ablating the second termλΣ_(i)F_(i)(θ_(i)−θ_(S,i))², excluding the EWC loss. Thus, the plotwithout regularization 1002 depicts a rapid change in weights withoutany regularization, which is implemented by setting λ=0. The plotwithout regularization 1002 shows a substantial deviation from anoriginal weight over the course of just a few hundred iterations. As aresult, images generated using techniques corresponding to the plotwithout regularization 1002 (e.g., without the EWC loss) include resultsthat are overfitted and are near regenerations of example images.

In contrast, the plot with regularization 1004 includes the second termλΣ_(i)F_(i)(θ_(i)−θ_(S,i))² that is used to iteratively calculate theEWC loss during target adaptation. The plot with regularization 1004shows that weights assigned to various parameters change slowly in earlyiterations of training the machine learning model 112. Such gradualchanges results in an increase of EWC loss over time. In addition, asthe EWC loss gradually saturates, the weights become unchanged, which isshown as the plot with regularization 1004 approaches a convergencepoint. Thus, by adapting few-shot images gradually over time, e.g.,using an applied EWC loss, values associated with the original weightsare altered while an overall amount of diversity of the weightedparameters is preserved.

Example of Transforming an Input Image into a Target Style Using aMachine Learning Model Trained for Few-Shot Image Generation ViaSelf-Adaptation

FIG. 11 depicts an example 1100 of images that are used in processes forusing a machine learning model trained to transform source images usingfew-shot image generation via self-adaptation, according to certainaspects of this disclosure. In particular, FIG. 11 depicts a simplifiedexample 1100 of a diagram of using a trained machine learning model fortransforming an image (e.g., input image 1104) into a target style(e.g., of target image 1106) using few-shot image generation viaself-adaptation. Operations described with respect to FIG. 11 transformimages using an image processing system (e.g., image processing system108). Image processing system 108 uses a trained machine learning model(e.g., machine learning model 112) to generate a target image 1106 in atarget style. One or more processing devices (e.g., computingenvironment 100) perform operations depicted in FIG. 11 by executingsuitable program code (e.g., machine learning model 112). Forillustrative purposes, example 1100 is described with reference tocertain examples depicted in the figures, however, other implementationsare possible.

In the example 1100 shown in FIG. 11 , a client device 1102 accesses anapplication, e.g., via a GUI 1110. The client device 1102 communicateswith one or more remote computing devices (e.g., image database 106 orimage processing system 1108) using a network connection (e.g., via datanetwork 104). In this example, the client device 1102 transmits arequest to transform an input image 1104 to the image processing system1108 using GUI 1110. The image processing system 1108 executes a machinelearning model (e.g., machine learning model 112) that is trained totransform the input image 1104 into a target style (e.g., of targetimage 1106). The client device 1102 and the image processing system 1108include all of the capabilities described above with respect to FIG. 1and the client devices 102 and the image processing systems 108,respectively.

In the example 1100, the client device 1102 accesses an application byexecuting suitable program code for the operations described herein. Forinstance, the client device 1102 accesses the application by launchingit using the GUI 1110. In some examples, the GUI 1110 includes theapplication. Further, the application includes user-selectable optionsthat correspond to one or more desirable target styles. In someexamples, a user selects an input image (e.g., a photograph) fortransformation. Further, in some examples, the user selects the inputimage and a desired target style for the transformation.

In some examples, the GUI 1110 includes user-selectable options. Forinstance, some GUIs 1110 include one or more icons, buttons, searchbars, checkboxes, dropdowns, lists, menus, sliders, any other GUIelements capable of receiving a user input, or a combination of these.In one example, the GUI 1110 allows the user to separately manipulateone or more image parameters. For example, the GUI 1110 includesuser-selectable options for separate image parameters such as a color,filter, resolution, size, texture, brightness, another suitable imageparameter, or any other suitable image settings. In some examples, theseuser-selectable options allows a user to refine image parameterscorresponding to the target style. Further, in some examples, theuser-selectable options allow a user to modify (e.g., adjust orotherwise alter) an available (e.g., a user-selected or selectable)target style.

In some examples, the application includes a camera or video-recordingdevice. For example, the user takes photographs or records video contentusing the application. In some examples, the user accesses previouslyrecorded images (e.g., photographs, drawings, videos) using theapplication or another suitable application. In some examples, thepreviously-recorded image content includes one or more photographs,videos, vlogs, audiovisual messages, audiovisual clips, image files,drawings, movies, graphics, social media content (e.g., social mediastories), GIFs, or another suitable form of multimedia content. In someexamples, the user selects the target style from among a predeterminedlist of target styles.

In one example, the user has captured an image that includes aself-portrait (e.g., a selfie). The user enhances the selfie using theapplication. For example, the user selects an option to transform theselfie into a target style. The user uses the application and/or GUI1110 to access a list of available target styles (e.g., via a menu)associated with a desired target style. In one example, the user desiresto transform the selfie into an emoji in a particular artistic style andshe selects an option for “emoji” from a menu of artistic target styles.In some examples, the user selects an option for “emoji” from among alist of available artistic styles for emoticons (e.g., a sub-menu). Inthis example, the sub-menu includes emoticons such as emojis, Bitmojis™or smileys, etc. In some examples, various features or parameters of aselected target style are retrieved from a remote library of targetstyles.

Continuing with the example 1100, the image processing system 1108receives the request from the client device 1102. The image processingsystem 1108 responsively executes suitable program code (e.g., machinelearning model 112) to transform the input image 1104 into the targetimage 1106. The image processing system 1108 provides the input image1104 to the machine learning model 112. The machine learning model 112is trained to transform the input image 1104 into the target style.

In some examples, the machine learning model 112 transforms the inputimage 1104 into the target image 1106 that includes the target style. Inthis example, the target image 1106 includes various features that areassociated with the input image 1104. For instance, the input image 1104and the target image 1106 both share similar features such as adirectionality of combed hair, eyes having dark irises in contrast withbright sclerae, smile lines, and a pronounced chin. These sharedfeatures combine to produce a target image 1106 that is recognizable ashaving an overall appearance that is associated with the input image1104. In addition to these common features, the target image 1106 alsoreflects a trained model that is capable of generating varied images,while maintaining diversity and avoiding over-fitting.

For instance, the target image 1106 reflects a result from a machinelearning model trained to implement learned aesthetics that areassociated with the target domain. Specifically, the generated targetimage 1106 includes various changes to facial features such as analtered facial angle, exaggerated nose, reshaped eyes, more pronouncedeyebrows, and de-emphasized ears. The target image 1106 also reflectschanges to the overall aesthetic, e.g., by changing the facialexpression of the input image 1104 from an open smile to a closed smile,while selectively omitting certain smile lines and/or crow's feet,thereby creating a more youthful but slightly less jovial appearance.Thus, the target image 1106 depicts a diverse set of features of thetarget style, while avoiding certain pitfalls of merely re-generatedinput images. In some examples, the target style is determined usingimage information associated with the input image 1104 (e.g., auser-selected target style). Further, in some examples, the target styleis determined by the image processing system 1108 and/or using a defaulttarget style.

The machine learning model 112 transforms the input image 1104 into thetarget image 1106 using one or more parameters associated with thetarget style. The machine learning model 112 performs the transformationbased on the training techniques discussed herein, modifying one or morefeatures of the input image 1104 to match the target style (e.g., usingparameters of the target style). The machine learning model 112 providesthe generated target image 1106 to image processing system 1108. Theimage processing system 1108 transmits the target image 1106 to theclient device 1102, e.g., via data network 104.

Examples of Computational Improvements Facilitated by Few-Shot ImageGeneration Via Self-Adaptation

Certain aspects described above, with respect to FIGS. 1-11 , enableimproved image generation that preserves diverse features using lesstraining data than, for example, conventional image generationtechniques. For instance, FIG. 12 depicts examples 1200 of conventionaltechniques that involve operations avoided by certain aspects of thisdisclosure. For example, some existing models require tedious manualdesigns that are less effective in extremely low-data cases. Theexamples 1200 shown in FIG. 12 illustrate results produced by some ofthese less effective models, which include Neural Style Transfer (NST)images 1204, Batch Statistics Adaptation (BSA) images 1206, and MineGANimages 1208.

In this example, the NST images 1204, BSA images 1206, and MineGANimages 1208 were generated using few-shot images 1202. The few-shotimages 1202 are substantially the same as few-shot images 802, whichwere discussed above, with respect to FIG. 8 . But in this example, theNST images 1204, BSA images 1206, and MineGAN images 1208 were generatedusing a trained NST model, a trained BSA model, and a trained MineGANmodel, respectively. And in this example, an image was randomly selectedfrom the few-shot images 1202 to generate each of the NST images 1204,BSA images 1206, and MineGAN images 1208, for example, using a randomlysampled image from a source domain (e.g., a source image 606 from acorresponding source domain).

The trained NST model separates and recombines image content usingparameters derived from semantic information, while the trained BSAmodel and trained MineGAN model both focus on adapting models from asource domain to a target domain by introducing additional parameters.For instance, the BSA model includes additional batch norm layers, e.g.,by modifying a BigGAN generator to learn new parameters during anadaptation. The MineGAN model adds a small mining network to aProgressive GAN generator, fine-tuning the small mining networkseparately from a joint fine-tuning operation of the small miningnetwork and the Progressive GAN generator. Each of the trained NSTmodel, trained BSA model, and trained MineGAN model was implementedusing a StyleGAN framework.

In some examples, the StyleGAN framework includes normalizing andmapping inputs to an intermediate latent space. The intermediate latentspace controls the StyleGAN generator using adaptive instancenormalization (AdaIN) for each convolutional layer. Further, theStyleGAN framework includes adding Gaussian noise after each convolutionand before evaluating nonlinearity. And in some examples, the StyleGANgenerator generates stochastic details for images using noise inputsthat are directly input into the generator.

But in this example, the trained NST model, trained BSA model, andtrained MineGAN model described above generated the NST images 1204, BSAimages 1206, and MineGAN images 1208 using a source domain of real facesand the examples shown in the few-shot images 1202. The examples 1200depict some examples of results that reflect the computationaldeficiencies associated with these conventional techniques. Forinstance, the trained NST model generated NST images 1204 thattransferred certain global colors and textures of the target examples,but the NST images 1204 include cluttered features and do not capturemany high-level characteristics (e.g., geometric shapes) of the targetstyle. Similarly, the trained BSA model generated BSA images 1206 thatappear blurry and out-of-focus. Additionally, the BSA images 1206illustrate a common failure of mode collapse, e.g., depicting severalresulting images that are substantially, visually similar. Further, thetrained MineGAN model generated MineGAN images 1208 that are over-fittedand generally less diverse. For instance, the MineGAN images 1208include several results that include a near re-generation of certainfeatures from the given examples in the few-shot images 1202.

Example of a Computing System for Implementing Certain Aspects

Any suitable computing system can be used for performing the operationsdescribed herein. FIG. 13 depicts an example of a computing system 1300that performs certain operations described herein, according to certainaspects of this disclosure. In some aspects, the computing system 1300executes image processing system 108 of FIG. 1 . In other aspects,separate computing systems having devices similar to those depicted inFIG. 13 (e.g., a processor, a memory, etc.) separately execute parts ofthe image processing system 108.

The example of a computing system 1300 includes a processor 1302communicatively coupled to one or more memory devices 1304. Theprocessor 1302 executes computer-executable program code 1316 stored inmemory device 1304, accesses information (e.g., program data 1318)stored in the memory device 1304, or both. Examples of the processor1302 include a microprocessor, an application-specific integratedcircuit (“ASIC”), a field-programmable gate array (“FPGA”), or any othersuitable processing device. Processor 1302 includes any number ofprocessing devices, including a single processing device.

The memory device 1304 includes any suitable non-transitorycomputer-readable medium for storing data, program code, or both. Acomputer-readable medium includes any electronic, optical, magnetic, orother storage device capable of providing a processor withcomputer-readable instructions or other program code. Non-limitingexamples of a computer-readable medium include a magnetic disk, a memorychip, a ROM, a RAM, an ASIC, optical storage, magnetic tape or othermagnetic storage, or any other medium from which a processing devicethat reads instructions. The instructions includes processor-specificinstructions generated by a compiler or an interpreter from code writtenin any suitable computer-programming language, including, for example,C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, andActionScript.

The computing system 1300 also includes a number of external or internaldevices, such as input or output devices. For example, the computingsystem 1300 is shown with one or more input/output (“I/O”) interfaces1308. An I/O interface 1308 receives input from input devices (e.g.,input device 1312) or provide output to output devices. One or morebuses 1306 are also included in the computing system 1300. The bus 1306communicatively couples one or more components of a respective one ofthe computing system 1300.

The computing system 1300 executes program code 1316 that configures theprocessor 1302 to perform one or more operations described herein. Forexample, the program code 1316 includes the machine learning model 112(including the adaptation engine 114, the parameter analysis engine 116,and the target generation engine 118), the pre-trained generative model110, or other suitable applications to perform operations describedherein. The program code 1316 resides in the memory device 1304 or anysuitable computer-readable medium that is executable by the processor1302 or another suitable processor. In additional or alternativeaspects, the program code 1316 described above is stored in one or moreother memory devices accessible via data network 104.

The computing system 1300 also includes a network interface device 1310.The network interface device 1310 includes any device or group ofdevices suitable for establishing a wired or wireless data connection toone or more data networks. Non-limiting examples of the networkinterface device 1310 include an Ethernet network adapter, a modem,and/or the like. The computing system 1300 is able to communicate withone or more other computing devices via data network 104 using thenetwork interface device 1310.

In some aspects, the computing system 1300 also includes presentationdevice 1314. A presentation device 1314 includes any device or group ofdevices for providing visual, auditory, or other suitable sensoryoutput. Non-limiting examples of presentation device 1314 include atouchscreen, a monitor, a speaker, a separate mobile computing device,etc. In some aspects, presentation device 1314 includes a remoteclient-computing device, such as client device 102, that communicateswith computing system 1300 using one or more data networks (e.g., datanetwork 104) described herein. Other aspects omit presentation device1314.

General Considerations

While the present subject matter has been described in detail withrespect to specific aspects thereof, it will be appreciated that thoseskilled in the art, upon attaining an understanding of the foregoing,may readily produce alterations to, variations of, and equivalents tosuch aspects. Numerous specific details are set forth herein to providea thorough understanding of the claimed subject matter. However, thoseskilled in the art will understand that the claimed subject matter maybe practiced without these specific details. In other instances,methods, apparatuses, or systems that would be known by one of ordinaryskill have not been described in detail so as not to obscure claimedsubject matter. Accordingly, this disclosure has been presented for thepurpose of providing examples rather than limitation, and does notpreclude the inclusion of such modifications, variations, and/oradditions to the present subject matter as would be readily apparent toone of ordinary skill in the art.

Unless specifically stated otherwise, it is appreciated that throughoutthis specification discussions utilizing terms such as “processing,”“computing,” “calculating,” “determining,” and “identifying” or the likerefer to actions or processes of a computing device, such as one or morecomputers or a similar electronic computing device or devices, thatmanipulate or transform data represented as physical electronic ormagnetic quantities within memories, registers, or other informationstorage devices, transmission devices, or display devices of thecomputing platform. The use of “adapted to” or “configured to” herein ismeant as open and inclusive language that does not foreclose devicesadapted to or configured to perform additional tasks or steps.Additionally, the use of “based on” is meant to be open and inclusive,in that a process, step, calculation, or other action “based on” one ormore recited conditions or values may, in practice, be based onadditional conditions or values beyond those recited. Headings, lists,and numbering included herein are for ease of explanation only and arenot meant to be limiting.

Aspects of the methods disclosed herein may be performed in theoperation of such computing devices. The systems discussed herein arenot limited to any particular hardware architecture or configuration. Acomputing device includes any suitable arrangement of components thatprovide a result conditioned on one or more inputs. Suitable computingdevices include multi-purpose microprocessor-based computer systemsaccessing stored software that programs or configures the computingsystem from a general purpose computing apparatus to a specializedcomputing apparatus implementing one or more aspects of the presentsubject matter. Any suitable programming, script, or other type oflanguage or combinations of languages may be used to implement theteachings herein in software to be used in programming or configuring acomputing device. The order of the blocks presented in the examplesabove can be varied—e.g., blocks can be re-ordered, combined, and/orbroken into sub-blocks. Certain blocks or processes can be performed inparallel.

The invention claimed is:
 1. A method in which one or more processingdevices perform operations comprising: receiving a request to transforman input image into a target image; providing the input image to amachine learning model trained to adapt images, wherein the machinelearning model is trained based on an adapted source domain, the adaptedsource domain comprising source images that have been modified based onrates of change of target parameters associated with a target style in atarget domain, wherein the machine learning model is trained by:generating an adapted generative model using the target parametersassociated with the target style; applying weights to the targetparameters based on their respective rates of change; and generating thetarget image by applying the machine learning model to the input image,wherein the target image comprises the target style.
 2. The method ofclaim 1, wherein the machine learning model is trained by: determiningthe rates of change of target parameters associated with the targetstyle in the target domain comprises: selecting a subset of the sourceimages from a source domain; identifying the target parametersassociated with the target style; and adapting a pre-trained generativemodel by applying an adversarial loss to the target parametersassociated with the target style, the pre-trained generative modelconfigured to generate the adapted source domain.
 3. The method of claim1 wherein applying the weights comprises: convolving, iteratively, aplurality of layers using a pre-trained generative model configured togenerate the adapted source domain and an adapted generative model; anddetermining an average rate of change of weights for each of the targetparameters at each of the plurality of layers.
 4. The method of claim 3,wherein determining the average rate of change of the weights for eachof the target parameters at each of the plurality of layers comprises:computing, for each of the plurality of layers, the average rate ofchange of the weights using an expression:${\bigtriangleup = {\frac{1}{N}{\sum}_{i}\frac{❘{\theta_{G^{\prime},i} - \theta_{G,i}}❘}{\theta_{G,i}} \times 100\%}},$wherein G represents a generator for the pre-trained generative model,G′ represents a generator for the adapted generative model, N representsa number of the target parameters, θ_(G,i) represents an i-th parameterin the pre-trained generative model, and G′ represents the adaptedgenerative model.
 5. The method of claim 1, wherein generating theadapted generative model further comprises applying an adversarial lossto the target parameters associated with the target style to apre-trained generative model using a deep convolutional generativeadversarial network (DCGAN), wherein the pre-trained generative model isconfigured to generate the adapted source domain, and wherein applyingthe adversarial loss to the target parameters comprises: computing theadversarial loss using an expression:${L_{adv} = {{\min\limits_{G}\max\limits_{D}{\varepsilon_{x \sim {P_{data}(x)}}\left\lbrack {\log{D(x)}} \right\rbrack}} + {\varepsilon_{z \sim {P_{z}(z)}}\left\lbrack {\log\left( {1 - {D\left( {G(z)} \right)}} \right)} \right\rbrack}}},$and wherein G represents a generator, D represents a discriminator,P_(data)(x) represents a distribution of noise associated with a subsetof source images, and P_(z)(z) represents a distribution of noiseassociated with the target parameters.
 6. The method of claim 5, whereinthe adapted source domain is generated in part by: generating aplurality of weighted parameters by applying a weight to each of thetarget parameters based on the rates of change; and wherein the adaptedsource domain is generated in part by: regularizing the plurality ofweighted parameters to create an importance factor for each of theplurality of weighted parameters; and applying the plurality of weightedparameters to the source domain based on the importance factors.
 7. Themethod of claim 6, wherein regularizing the plurality of weightedparameters to create the importance factor for each of the plurality ofweighted parameters comprises: generating importance data for each ofthe plurality of weighted parameters, wherein the importance datacomprises Fisher information F and is computed using an expression:${F = {E\left\lbrack {- {\frac{\partial^{2}}{\partial\theta_{s}^{2}}{\mathcal{L}\left( {X{❘\theta_{s}}} \right)}}} \right\rbrack}},$and wherein θ_(S) represents the plurality of weighted parameters, and

(X|θ_(s)) represents a log-likelihood function that is configured toprovide an equivalent of a binary cross-entropy loss using an output ofthe discriminator D.
 8. The method of claim 7, wherein the machinelearning model is further trained by: determining an elastic weightconsolidation (EWC) loss using the Fisher information F associated witheach of the plurality of weighted parameters; and transforming thesource domain into the adapted source domain using the EWC loss.
 9. Themethod of claim 8, wherein determining the EWC loss comprises: computingthe EWC loss by adding a regularization loss configured to penalize arate of change associated with the target parameters and using anexpression:L _(adapt) =L _(adv)+λΣ_(t) F _(t)(θ_(t)−θ_(S,t))², wherein L_(adapt)represents the EWC loss, and λ represents a regularization weight.
 10. Asystem comprising: one or more processing devices; and a non-transitorycomputer-readable medium communicatively coupled to the one or moreprocessing devices and storing instructions, wherein the one or moreprocessing devices are configured to execute the instructions andthereby perform operations comprising: accessing training datacomprising a source domain that includes a plurality of source imagesand a target domain that includes a limited number of artistic images,wherein the target domain comprises a target style; and generating,using a pre-trained generative model, an adapted source domaincomprising a plurality of adapted images, wherein the plurality ofadapted images comprises the target style, and wherein the adaptedsource domain is generated by: determining a rate of change for each ofa plurality of parameters associated with the target style; generatingan adapted generative model using the plurality of parameters associatedwith the target style; generating a plurality of weighted parameters byapplying a weight to each of the plurality of parameters based on therate of change; applying the plurality of weighted parameters to thesource domain; and outputting a trained machine learning modelconfigured to generate, from an input image, a target image comprisingthe target style.
 11. The system of claim 10, wherein applying a weightto each of the plurality of parameters based on the rate of changecomprises: convolving, iteratively, a plurality of layers using thepre-trained generative model and an adapted generative model; anddetermining an average rate of change of weights for each of theplurality of parameters at each of the plurality of layers, whereindetermining the average rate of change of the weights comprises:computing, for each of the plurality of layers, the average rate ofchange of the weights using an expression:${\bigtriangleup = {\frac{1}{N}{\sum}_{t}\frac{❘{\theta_{G^{\prime},i} - \theta_{G,i}}❘}{\theta_{G,i}} \times 100\%}},$wherein G represents the pre-trained generative model, G′ represents theadapted generative model, N represents a number of the plurality ofparameters, θ_(G,i) represents an i-th parameter in the pre-trainedgenerative model, and G′ represents the adapted generative model. 12.The system of claim 10, wherein the pre-trained generative modelcomprises a DCGAN, and the operations further comprising: generating anadapted generative model by applying an adversarial loss to theplurality of parameters associated with the target style to thepre-trained generative model using the DCGAN, wherein applying theadversarial loss to the plurality of parameters comprises: computing theadversarial loss using an expression:${L_{adv} = {{\min\limits_{G}\max\limits_{D}{\varepsilon_{x \sim {P_{data}(x)}}\left\lbrack {\log{D(x)}} \right\rbrack}} + {\varepsilon_{z \sim {P_{z}(z)}}\left\lbrack {\log\left( {1 - {D\left( {G(z)} \right)}} \right)} \right\rbrack}}},$ and wherein G represents a generator, D represents a discriminator,P_(data)(x) represents a distribution of noise associated with a subsetof source images, and P_(z)(z) represents a distribution of noiseassociated with the plurality of parameters.
 13. The system of claim 12,wherein the adapted source domain is generated in part by: regularizingthe plurality of weighted parameters to create an importance factor foreach of the plurality of weighted parameters, wherein regularizing theplurality of weighted parameters to create the importance factor foreach of the plurality of weighted parameters comprises: generatingimportance data for each of the plurality of weighted parameters,wherein the importance data comprises Fisher information F and iscomputed using an expression:${F = {E\left\lbrack {- {\frac{\partial^{2}}{\partial\theta_{s}^{2}}{\mathcal{L}\left( {X{❘\theta_{s}}} \right)}}} \right\rbrack}},$ and wherein θ_(S) represents the plurality of weighted parameters, and

(X|θ_(s)) represents a log-likelihood function that is configured toprovide an equivalent of a binary cross-entropy loss using an output ofthe discriminator D; and applying the plurality of weighted parametersto the source domain based on the importance factors.
 14. The system ofclaim 13, the operations further comprising: determining an EWC lossusing the Fisher information F associated with each of the plurality ofweighted parameters; computing the EWC loss by adding a regularizationloss configured to penalize a rate of change associated with theplurality of parameters and using an expression:L _(adapt) =L _(adv)+λΣ_(t) F _(t)(θ_(t)−θ_(S,t))², wherein L_(adapt)represents the EWC loss, and λ represents a regularization weight; andtransforming the source domain into the adapted source domain using theEWC loss.
 15. A method in which one or more processing devices performoperations comprising: accessing training data comprising a sourcedomain that includes a plurality of source images and a target domainthat includes a limited number of artistic images, wherein the targetdomain comprises a target style; a step for generating an adapted sourcedomain comprising a plurality of adapted images, wherein the pluralityof adapted images comprises the target style; training a machinelearning model on the plurality of adapted images, the adapted sourcedomain comprising source images that have been modified based on ratesof change of target parameters associated with the target style in thetarget domain, wherein the machine learning model is trained by:generating an adapted generative model using the target parametersassociated with the target style; applying weights to the targetparameters based on their respective rates of change; and outputting themachine learning model configured to apply the target style to an inputimage using the adapted source domain.
 16. The method of claim 15wherein determining the rate of change for each of the parameterscomprises: selecting a subset of source images from the source domain;and adapting the pre-trained generative model by applying an adversarialloss to the target parameters associated with the target style.
 17. Themethod of claim 16, the operations further comprising: generating anadapted generative model using the target parameters associated with thetarget style by: convolving a plurality of layers using the pre-trainedgenerative model and an adapted generative model; and determining anaverage rate of change of weights for each of the target parameters,wherein determining the average rate of change of the weights for eachof the target parameters comprises: computing, for each of the pluralityof layers, the average rate of change of the weights using anexpression:${\bigtriangleup = {\frac{1}{N}{\sum}_{t}\frac{❘{\theta_{G^{\prime},i} - \theta_{G,i}}❘}{\theta_{G,i}} \times 100\%}},$ and wherein G represents the pre-trained generative model, G′represents the adapted generative model, N represents a number of thetarget parameters, θ_(G,i) represents an i-th parameter in thepre-trained generative model, and G′ represents the adapted generativemodel.
 18. The method of claim 16, wherein the pre-trained generativemodel comprises a DCGAN, and the operations further comprising:generating an adapted generative model by applying an adversarial lossto weighted parameters associated with the target style to thepre-trained generative model using the DCGAN, wherein applying theadversarial loss to the weighted parameters comprises: computing theadversarial loss using an expression:${L_{adv} = {{\min\limits_{G}\max\limits_{D}{\varepsilon_{x \sim {P_{data}(x)}}\left\lbrack {\log{D(x)}} \right\rbrack}} + {\varepsilon_{z \sim {P_{z}(z)}}\left\lbrack {\log\left( {1 - {D\left( {G(z)} \right)}} \right)} \right\rbrack}}},$ and wherein G represents a generator, D represents a discriminator,P_(data)(x) represents a distribution of noise associated with a subsetof source images, and P_(z)(z) represents a distribution of noiseassociated with the weighted parameters.
 19. The method of claim 18,wherein the adapted source domain is generated in part by: regularizingthe weighted parameters to create an importance factor for each of theweighted parameters, wherein regularizing the weighted parameters tocreate the importance factor for each of the weighted parameterscomprises: generating importance data for each of the weightedparameters, wherein the importance data comprises Fisher information Fand is computed using an expression:${F = {E\left\lbrack {- {\frac{\partial^{2}}{\partial\theta_{s}^{2}}{\mathcal{L}\left( {X{❘\theta_{s}}} \right)}}} \right\rbrack}},$ and wherein θ_(S) represents the weighted parameters, and

(X|θ_(s)) represents a log-likelihood function that is configured toprovide an equivalent of a binary cross-entropy loss using an output ofthe discriminator D; and applying the weighted parameters to the sourcedomain based on the importance factors.
 20. The method of claim 19, theoperations further comprising: determining an EWC loss using the Fisherinformation F associated with each of the weighted parameters; computingthe EWC loss by adding a regularization loss configured to penalize arate of change associated with the weighted parameters and using anexpression:L _(adapt) =L _(adv)+λΣ_(t) F _(t)(θ_(t)−θ_(S,t))², wherein L_(adapt)represents the EWC loss, and λ represents a regularization weight; andtransforming the source domain into the adapted source domain using theEWC loss.